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Abstract 

Homework is transforming at a rapid rate with continuous advances in educational technology. 
Computer-based homework, in particular, is gaining popularity across a range of schools, with 
little empirical evidence on how to optimize student learning. The current aim was to test the 
effects of different types of feedback on computer-based homework. In the study, middle school 
students completed a computer-based pretest, homework assignment, and posttest containing 
challenging algebraic problems. On the homework assignment, students were assigned to 
different feedback conditions. In Experiment 1 (N= 103), students received no feedback or 
correct-answer feedback after each problem. In Experiment 2 (N= 143), students received (1) no 
feedback, (2) correct-answer feedback, (3) try-again feedback, or (4) explanation feedback after 
each problem. For students with low prior knowledge, feedback resulted in better posttest 
performance than no feedback. However, students with high prior knowledge learned just as 
much whether they received feedback or not. Results suggest the provision of basic feedback on 
computer-based homework can benefit novice students’ mathematics learning. 


Key words: feedback, problem solving, computer-based homework, mathematics learning 
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Providing feedback on computer-based algebra homework in middle-school classrooms 

1. Introduction 

Modern advances in educational technology and increasing access to computers gives 
teachers a wide range of tools for assigning homework and assessing student progress. Intelligent 
tutor systems and computer-based homework are quickly gaining popularity and prevalence in 
classrooms across the world. One of the bedrocks of these systems is the availability of 
individualized, just-in-time feedback (Ma et al., 2014). Indeed, many researchers have attributed 
the effectiveness of computer tutors and computer-based homework, at least in part, to greater 
frequency and immediacy of feedback (e.g., Azevedo & Bernard, 1995; Martin et al., 2007). The 
goal of the current research was to experimentally evaluate the effects of feedback on computer- 
based algebra homework for middle school students with varying prior knowledge. 

In general, research supports the use of feedback as meta-analyses continue to show that, 
on average, feedback has positive effects on learning outcomes relative to no feedback (Alfieri et 
al., 2011; Hattie & Timperley, 2007). However, there is considerable variability; feedback helps 
in some cases, but not others (see Hattie & Gan, 2011). A growing body of evidence suggests 
that some of the variability in feedback effects is due to students’ prior knowledge (e.g., Fyfe, 
Rittle-Johnson & DeCaro, 2012; Fyfe & Rittle-Johnson, 2016a, 2016b; Gielen et al., 2010; 
Krause et al., 2009). Specifically, feedback often has strong, positive effects for students with 
lower prior knowledge, but neutral or even negative effects for students with higher prior 
knowledge. For example, in Fyfe et al. (2012), second- and third-grade students solved novel 
math problems with or without feedback in a one-on-one tutoring context. Students with low 
prior knowledge on a pretest benefited from feedback. In contrast, students with higher prior 
knowledge on a pretest learned less when immediate, corrective feedback was provided. 
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There are several potential reasons why feedback may result in lower learning than no 
feedback. For example, feedback may reduce mindful processing of the task if students become 
over-reliant on the feedback message (e.g., Schmidt & Bjork, 1992), feedback may draw 
attention to the self rather than the task and elicit affective reactions (e.g., I must not be smart) 
that interfere with learning (Kluger & DeNisi, 1996), or feedback may overload cognitive 
resources simply by providing additional infonnation that needs integrated with the student’s 
prior knowledge (Sweller, van Merrienboer, & Paas, 1998). Given these potential consequences, 
more work is needed to identify the conditions under which feedback is effective. 

One learning context in which the provision of feedback is particularly relevant is 
computer-based homework. As access to computers increases, so too does the prevalence of 
systems that allow students to complete their homework online. These systems are gaining 
traction in K-12 schools as more schools in the U.S. are adopting one-to-one computing 
programs, which supply each student with his/her own laptop for classroom work (Bebell & Kay, 
2010). The goal of the current study was to experimentally evaluate the effects of feedback using 
a particular system, ASSISTments.org (Heffernan & Heffernan, 2014). ASSISTments is a 
computer system that can provide scaffolds and feedback to assist student learning. The use of 
computer-based homework offers several advantages for understanding the effects of feedback. 

First, computer-based homework provides an ecologically valid context in which to 
evaluate the role of feedback on student learning. Many prior studies that have experimentally 
evaluated the effect of feedback have been conducted in laboratory contexts in the presence of a 
researcher (e.g., Fyfe & Rittle-Johnson, 2016a, 2016b; Krause et ah, 2009). Computer-based 
homework provides a means to experimentally test the effects of feedback in an authentic 
learning setting on homework assignments given to students by their teachers. 
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Second, computer-based homework represents a learning setting that may reduce the 
potential negative effects of feedback (see Fyfe & Rittle-Johnson, 2016b). As mentioned above, 
one condition under which feedback may hinder learning is when it draws attention to the self 
and evokes evaluations of one’s self or abilities (Kluger & DeNisi, 1996). Computer-generated 
feedback is often viewed as a less evaluative source of information than person-generated 
feedback (Karabenick & Knapp, 1988), and may help decrease attention on the self. Computer- 
based homework may also reduce cognitive overload by giving students control over when and 
how they process the feedback. For example, students can choose whether and how long to study 
the feedback message, and they can choose when they are ready to move on. 

Third, computer-based homework represents a flexible system for evaluating different 
feedback types. Dempsey, Driscoll and Swindell (1993) outlined a hierarchy of feedback types 
based on the information provided: 

1. No feedback : provides no information about the student’s response. 

2. Verification feedback: informs the student if the response is correct or incorrect. 

3. Correct-answer feedback: informs the student what the correct response is. 

4. Elaborated feedback: provides some explanation for why a response is correct or 
incorrect or allows the student to review part of the instruction. 

5. Try-again feedback: informs the student if the response is correct or incorrect and 
allows one or more additional attempts to try again. 

One possibility is that providing feedback with more information will have positive effects for 

both low- and high-knowledge students. Indeed, one of the advantages of computer-based 

homework is the ability to provide second attempts, hints, and explanations to guide student 

learning. There is some consensus that effective feedback should go beyond verification and 

provide the correct answer (see Bangert-Drowns, et ah, 1991 and Kluger & DeNisi, 1996 for 

meta-analyses). But, the benefits of providing additional information are less clear (Mory, 2004). 
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Finally, ASSISTments in particular provides a unique platform for conducting 
educational research that maintains rigorous experimental control. In addition to providing a 
learning a tool for students and teachers, ASSISTments supports researchers in creating 
randomized controlled experiments (Heffeman & Heffernan, 2014). It includes a building 
interface that allows researchers and teachers to write content and create assignments that vary 
on one or more dimensions (e.g., the presence versus absence of feedback). It then allows 
students within the same classroom to be randomly assigned to different assignments. This type 
of system is key for conducting experimental research in classroom settings. 

The current study tested the effects of feedback for middle school students solving 
algebraic equations on computer-based homework via the ASSISTments system. In Experiment 

1, students were assigned to receive correct-answer feedback or no feedback during their 
homework assignment. Experiment 2 included two additional feedback conditions: explanation 
feedback and try-again feedback. Based on previous research, feedback was predicted to interact 
with prior knowledge such that feedback would have a stronger, positive effect on learning and 
problem solving for students with low prior knowledge on the pretest. 

2. General Method 

2.1. Participants. In each experiment, the participants were students from public middle 
school classrooms in the U.S. whose teachers were using the ASSISTments system as part of 
their regular classroom experience and volunteered to participate. 

2.2. Materials. All materials were presented using ASSISTments.org. Problems were 
presented one at a time on the computer screen and students typed their response to each item. In 
each experiment, materials included a pretest, a homework assignment, and a posttest, all of 
which assessed students’ abilities to solve algebraic equations. There were four different 
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equation types: ax + b = c, b + ax = c, a(x + b) = c, and a(x + b) + c = d. Table 1 displays the 
items presented on the pretest and posttest. The homework assignment contained two worked 
examples at the beginning of the assignment to familiarize students with correct problem-solving 
solutions (see Figure 1 for an example). The worked examples demonstrated a step-by-step 
solution to each problem and provided the correct answers. The remaining problems on the 
homework were equations for the students to solve on their own. The problems were similar to 
those presented on the pretest and the problem types were presented in a mixed sequence. 


Table 1: Problems presented on the pretest and posttest in Experiment 1 and Experiment 2 



Experiment 1 


Experiment 2 



Pretest 

Posttest 

Pretest 

Posttest 

1. 

2x + 3 = 23 

5x + 6 = 46 

2x + 3 = 23 

5x+ 13 = 73 

2. 

10 + 5x = 30 

3 + 9x = 57 

10 + 5x = 30 

3 + 6x = 99 

3. 

3(x + 1) = 9 

6(x + 3) = 42 

3(x + 1) = 9 

7(x + 4) = 63 

4. 

7x+ 11 = 53 

8x + 7 = 31 

7(x + 3) + 2 = 51 

2(x + 3) + 4 = 16 

5. 

6 + 4x = 42 

4 + 2x = 36 

5(x + 3) + 14 = 64 

x/2 + 3 = 13 

6. 

8(x + 2) = 56 

7(x + 1) = 21 

8(x + 2) = 56 

4(x + 2) + 3(x + 2) = 35 

7. 

— 

x/2 + 3 = 13 

— 

x/9 + 31 = 34 

8. 

— 

2(x + 3) + 4 = 16 

— 

6(x + 4) + 2(x +4) = 48 


Note. In Experiment 1, problems 7 and 8 on the posttest were transfer problems. In Experiment 
2, problems 5 through 8 on the posttest were transfer problems. 


2.3. Design and Procedure. The experiments had a pretest-homework-posttest design. 
Students completed the pretest on computers during class or at home. Within three school days, 
students completed the homework assignment on their own. For the homework assignment, 
students were randomly assigned to one of several feedback conditions (described below). 
Finally, students completed the posttest. All teachers assigned the posttest the same day students 
finished the homework, but some had students complete it in class and others had students 
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complete it at home. All procedures were implemented in accord with the American 
Psychological Association guidelines for the ethical treatment of human participants. 

Figure 1: A worked example presented at the beginning of the homework assignment 

Example Problem 1 

Solve for x in the following equation: 

3x + 15 = 27 

Example Solution 

Here is one correct way to solve this problem. 

3x + 15 = 27 State the problem. 

3x + 15 - 15 = 27 - 15 Since 15 is added to 3x, subtract 15 from both sides of the equation. 

3x = 12 Simplify. 

3x _ 12 

3 3 Since x is multiplied by 3, divide both sides by 3. 

x = 4 Now x is alone on one side of the equation. The solution is 4. 

Type the solution in the box below. 


Type your answer below (mathematical expression): 

Submit Answer 


3. Experiment 1 

3.1. Participants. All students from one fifth-grade teacher’s classrooms and one sixth- 
grade teacher’s classrooms were invited to participate. Of their 151 students, 48 students were 
not included in the study as they did not complete all required sessions. The final sample 
contained 103 students (45 in fifth-grade and 58 in sixth-grade). 

3.2. Materials. The pretest contained six problems (a = .69, see Table 1). There were 
three different problem types and two of each type. Total scores (out of six) were calculated for 
each student. The homework assignment contained 18 problems to solve and included six 
problems of each type. The posttest contained eight problems (a = .85, see Table 1). The first six 
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learning items were isomorphic to the pretest problems. Total scores (out of six) were calculated 
for each student. The last two items were transfer problems that differed in structure from the 
homework problems. Children scored a 1 if they solved both items correctly. 

3.3. Design and Procedure. For the homework assignment, students were randomly 
assigned to one of two conditions: no-feedback (n = 58) or correct-answer feedback (n = 45). In 
the no-feedback condition, the homework instructions included: “You will not receive feedback 
about whether your answers are correct or incorrect. This allows you to work without 
interruption. This gives you the chance to process the problems and answers on your own.” 
During the assignment, students did not receive feedback. After submitting their answer, the 
computer provided an “answer recorded” message and students clicked a button to move. In the 
correct-answer feedback condition, the homework instructions included, “After you respond, 
you will receive feedback about whether your answer is correct or incorrect. This allows you to 
assess your progress. This gives you the chance to process the problems and the correct 
answers.” During the assignment, students received immediate, correct-answer feedback after 
each problem. If students typed the correct answer, a green check mark appeared with the word 
“Correct!” If students typed an incorrect answer, a red X appeared along with the words, “The 

correct answer is_” (with the correct answer filled in). Students could also obtain the correct 

answer by clicking on a button, but they were aware this would count as an incorrect response. 

3.4. Results. Two students did not finish the pretest, so their pretest scores were imputed 
using the expectation-maximization algorithm via the missing value analysis in SPSS (Schafer & 
Graham, 2002). To examine the impact of feedback and prior knowledge, regression analyses 
were used for each outcome measure. Each regression model included condition (coded 0 for no¬ 
feedback and 1 for feedback), pretest score (mean centered), and their interaction. 
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3.4.1. Pretest. On average, children solved 67% of the pretest problems correctly (SD = 
36%). Scores were statistically similar in the feedback (M= 70%, SD = 33%) and no-feedback 
(M= 64%, SD = 37%) conditions, t(101) = -0.86, p = .39. 

3.4.2. Homework. Overall, students did better on the homework problems (M= 76%, SD 
= 31%). The overall regression predicting homework scores was significant, F( 3, 99) = 37.66, p 
< .000, R 2 = .53. There was a positive effect of prior knowledge, B = 0.67, SE = 0.08, p < .000. 
Students with higher prior knowledge exhibited higher homework scores than students with 
lower prior knowledge. There was no main effect of condition, B = 5.01, SE = 4.29, p = .25, nor 
was there a condition by prior knowledge interaction, B = -0.12, SE = 0.12,/? = .33. In general, 
correct-answer feedback had little effect on the homework assignment. 

3.4.3. Posttest. Scores on the six learning items were moderate (M= 79%, SD = 31%), 
but significantly higher than pretest scores, i(102) = 4.77,/? < .001. The overall regression 
predicting posttest learning scores was significant, F( 3, 99) = 33.03,/? < .000, R 2 = .50. There 
was a significant, positive effect of prior knowledge, B = 0.70, SE = 0.08,/? < .000. The main 
effect of condition was not significant, B = 4.87, SE = 4.40 ,p = .27, but there was a significant 
condition by prior knowledge interaction, B = -0.29, SE = 0.13, /? = .02. 

To follow up the interaction, two regression analyses tested the effect of condition for 
students with lower and higher prior knowledge. Pretest scores were centered at one standard 
deviation below the mean in one model and one standard deviation above the mean in a separate 
model (see Aiken & West, 1991). As shown in Figure 2, for low-knowledge students, correct- 
answer feedback resulted in higher posttest scores than no-feedback. In contrast, for high- 
knowledge students, correct-answer feedback resulted in somewhat lower posttest scores than 
no-feedback. Indeed, the main effect of condition was positive and significant feedback for 
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students with lower prior knowledge, B = 15.27, SE = 6AQ,p = .02, and negative, but non¬ 
significant for students with higher prior knowledge, B = -5.53, SE = 6.20, p = .38. 

Perfonnance on the transfer problems was relatively low. Fifty-three percent of students 
solved both problems correctly. A logistic regression predicting the log of the odds of solving 
both problems correctly showed a positive, main effect of prior knowledge, B = 0.03, SE = 0.01, 
p= .001. However, the main effect of condition was not significant,/? = .96, nor was the 
condition by prior knowledge interaction,/? = .62. On these more difficult transfer problems, 
correct-answer feedback had a neutral effect for both low- and high-knowledge students. 



Note. Non-standardized regression coefficients are plotted at plus/minus one standard deviation 
from the mean. 
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3.4. Discussion. In Experiment 1, the hypothesis that feedback would interact with prior 
knowledge was supported. On the posttest learning items, immediate correct-answer feedback 
had strong, positive effects for low-knowledge students, but neutral effects for high-knowledge 
students. The results are consistent with previous research suggesting that the effects of feedback 
vary across students with different levels of domain-specific knowledge (e.g., Fyfe et ah, 2012; 
Krause et al., 2012). The current study extended this finding to middle school students solving 
algebra problems on a computer-based homework assignment via the ASSISTments system. 

The goal of Experiment 2 was to replicate and extend the results of Experiment 1 by 
including two additional feedback conditions. Specifically, students were assigned to receive no 
feedback, correct-answer feedback, explanation feedback, or try-again feedback. A second 
modification was to expand the sample to include seventh-grade students and increase the 
difficulty of the materials (e.g., include an additional problem type, extra transfer items). 

4. Experiment 2 

4.1. Participants. All students from two sixth-grade teachers’ classrooms and two 
seventh-grade teachers’ classrooms were invited to participate. Of their 160 students, 17 students 
were not included in the study as they did not complete all required sessions. The final sample 
contained 143 students (65 in sixth-grade and 78 in seventh-grade). Average age was 
approximately 12.1 years based on birth year (SD = 0.7) and 47% of students were female. 

4.2. Materials. The pretest contained six problems (a = .74, see Table 1). The homework 
assignment contained 12 or 16 problems (i.e., three or four of each type). Whether students 
solved 12 or 16 problems reflected natural variation in teacher preference as two teachers opted 
for the 16-problem assignment (n = 65 students) and two teachers requested a shorter 12-problem 
assignment (n = 78 students). Percent correct scores were calculated for each student based on 
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the number of items he or she was assigned. The posttest included eight problems (a = .66, see 
Table 1). The first four items were isomorphic to the pretest problems (i.e., learning) and the 
remaining four were challenge problems with novel problem structures (i.e., transfer). Total 
scores (out of 4) for each scale (i.e., learning and transfer) were calculated for each student. 

4.3. Design and Procedure. For the homework assignment, students were randomly 

assigned to one of four conditions: no-feedback (n = 25), correct-answer feedback (n = 44), 

explanation feedback (n = 41), or try-again feedback (n = 33). The no-feedback and correct- 

answer feedback conditions were identical to Experiment 1. The explanation and try-again 

conditions were very similar to the correct-answer feedback, except the feedback message to an 

incorrect response varied. In the explanation feedback condition, the feedback message included 

the correct answer, an explanation of why it was correct, and a worked example. For example, 

for 3x + 12 = 24, the feedback message stated: 

The correct answer is 4 because when x is 4 both sides of the equal sign have the same 
amount. Let’s plug 4 in for x and simplify to show that both sides have the same amount. 

3x+ 12 = 24 
3*4 + 12 = 24 
12+ 12 = 24 
24 = 24 

In the try-again feedback condition, the feedback message stated, “Sorry, try again._is not 

correct” (with the student’s answer filled in). Students could continue inputting responses until 
they entered the correct answer or they could obtain the correct answer by clicking on a button. 

4.4. Results. To examine the impact of feedback and prior knowledge, regression 
analyses were used for each outcome measure. Each model included condition, pretest score, and 
their interaction. Condition was coded using the set of Helmert contrasts. Specifically, we 
contrasted (1) no-feedback versus the three feedback conditions, (2) correct-answer feedback 
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versus the two feedback conditions with additional information, and (3) explanation feedback 
versus try-again feedback. Thus, each regression model included three condition variables, 
pretest score (mean centered), and three condition by pretest score interactions. 

4.4.1. Pretest. On average, children solved 85% of the pretest problems correctly ( SD = 
23%). Scores were similar across the four condition: no-feedback (M= 84%, SD = 20%), 
correct-answer feedback (M= 89%, SD = 20%), explanation feedback (M= 85%, SD = 24%), 
and try-again feedback ( M= 80%, SD = 28%), F( 1, 139) = 1.09 ,p = .36. 

4.4.2. Homework. Scores on the homework assignment were not normally distributed (M 
= 86%, Med= 92%, SD = 19%). Nearly half (43%) of students solved more than 90% correct. 
The logistic regression predicting the log of the odds of scoring above the median on the 
homework was significant, % (7, N= 143) = 15.00,= .04, but the only predictor to reach 
significance was the main effect of prior knowledge, B = 0.03, SE = 0.0\,p = .006. There were 
no main effects of condition or interactions, ps > .20. See Table 2 for the full regression results. 
Consistent with Experiment 1, feedback had little effect on the homework assignment. 


Table 2: Regression results predicting homework and posttest scores in Experiment 2 


Variable 

Homework: 
Above Median 

Posttest Learning: 
Above Median 

Posttest Transfer: 
Percent Correct 

Intercept 

0.41 (0.20)* 

0.28 (0.24) 

63.63 (2.72)*** 

Pretest Score 

0.03 (0.01)** 

0.06 (0.02)*** 

0.66 (0.12)*** 

Condition Contrasts 

(1) No-feedback vs feedback 

0.49 (0.50) 

2.08 (0.73)** 

4.55 (6.95) 

(2) Correct-answer vs other 

0.02 (0.45) 

0.41 (0.46) 

-8.10(6.10) 

(3) Explanation vs try-again 

0.33 (0.53) 

0.55 (0.59) 

-3.62 (7.42) 

Condition x Pretest Score 

(1) x pretest score 

0.01 (0.03) 

-0.11 (0.06) a 

-0.62 (0.35) a 

(2) x pretest score 

-0.01 (0.03) 

0.00 (0.02) 

0.26 (0.28) 

(3) x pretest score 

-0.03 (0.03) 

-0.02 (0.03) 

0.13 (0.29) 
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Note. Unstandardized coefficients are shown with standard errors in parentheses. Pretest scores 
were mean-centered. The estimates for homework and posttest learning scores are from logistic 
regression. p<.\9, p<. 05, p<.0\, < .001 

4.4.3. Posttest Learning Items. Scores on the four posttest learning items were also high 
and not normally distributed ( M= 86%, Med = 100%, SD = 20%), with a full 62% of students 
solving all four items correctly. The logistic regression predicting the log of the odds of scoring 
100% on the posttest learning items was significant, X <7, N= 143) = 37.49,/? < .000 (see Table 
2). There was a significant, positive effect of prior knowledge, B = 0.14, SE = 0.06, p = .02. 
There was also a significant effect of no-feedback versus feedback, B = 2.08, SE = 0.73, p = 

.004. Fewer children in the no-feedback condition (36%) solved all the learning items correctly 
compared to children who received feedback (68%). There were no significant differences 
between correct-answer feedback (66%) and the two other feedback conditions (69%), p = .37, 
or between the explanation feedback (66%) and try-again feedback conditions (72%), p= .35. A 
follow-up analysis revealed that each of the three feedback conditions was significantly better 
than the no-feedback condition,/is < .05. However, there was a marginal interaction between 
prior knowledge and the no-feedback vs. feedback contrast, B = -0.11, SE = 0.06 ,p = .08. 

To examine the interaction, pretest scores were centered at one standard deviation below 
the mean in one model and one standard deviation above the mean in a separate model. Results 
were consistent with Experiment 1. For low-knowledge students, there was a significant positive 
effect of feedback versus no-feedback, B = 4.55, SE = \.91 ,p= .02. A follow-up analysis 
revealed that each of the three feedback conditions was significantly better than the no-feedback 
condition, ps < .05. There were no differences between the contrasts comparing feedback types, 
ps > .15. In contrast, for high-knowledge students, there was a negative, but non-significant 
effect of feedback versus no-feedback, B = -0.38, SE = 1.14, p = .74, again with no differences 
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between the contrasts comparing feedback types, ps > .45. Overall, feedback resulted in greater 
mastery of the learning items than no feedback, but only for students with low prior knowledge. 
Further, the type of feedback provided did not change the magnitude of the effect. 

4.4.4. Posttest Transfer Items. Scores on the four transfer problems were somewhat low 
(M= 64%, SD = 34%), with only minor deviations from nonnality. The linear regression 
predicting percent correct on the posttest transfer items was significant, F(1 , 135) = 5.10. p < 
.000, R 2 = .21. There was a significant main effect of prior knowledge, B = 0.66, SE = 0.12,/? = 
.001, but no main effects of condition,/is > .15. However, there was a marginal interaction 
between prior knowledge and the no-feedback vs. feedback contrast, B = -0.62, SE = 0.35,/? = 
.08. See Figure 3 and Table 2 for the full regression results. 

To examine the interaction, pretest scores were centered at one standard deviation below 
the mean in one model and one standard deviation above the mean in a separate model. For low- 
knowledge students, there was a marginal positive effect of feedback versus no-feedback, B = 
18.98, SE = 10.48,/? = .07. A follow-up analysis revealed that correct-answer feedback was 
significantly better than no-feedback, p = .03, but the positive effects of explanation feedback,/? 
= .14, and try-again feedback,/? = .35, were not statistically significant. There were no effects of 
the contrasts comparing feedback types, ps > .10. However, for high-knowledge students, there 
was a negative, but non-significant effect of feedback versus no-feedback, B = -9.88, SE = 10.97, 
/? = .37, again with no differences between the contrasts comparing feedback types, ps > .75. 

4.4. Discussion. Experiment 2 was consistent with Experiment 1 and supported the 
hypothesis that feedback would interact with prior knowledge. For mastery on the posttest 
learning items, all three feedback types had positive effects for low-knowledge students, but 
neutral effects for high-knowledge students. Similarly, for transfer items, feedback tended to 
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result in higher transfer scores for low-knowledge students, but similar scores for high- 
knowledge students. The only piece of evidence in this experiment for a difference between the 
feedback types favored the correct-answer condition. Specifically, correct-answer feedback 
resulted in significantly higher transfer scores than no-feedback for low-knowledge students, but 
explanation-feedback and try-again feedback did not statistically differ from no feedback. These 
results suggest that in some cases, the presence versus absence of feedback may play a larger a 
role for student learning than the specific type of feedback provided. 



Note. Non-standardized regression coefficients are plotted at plus/minus one standard deviation 
from the mean. 
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5. Conclusion 

The goal of this study was to test the effects of feedback and prior knowledge on 
mathematics problem solving using the ASSISTments system. Middle school students solved a 
set of challenging algebraic equations on a computer-based homework assignment. Feedback had 
little impact on students’ homework perfonnance. However, on posttest learning items, feedback 
interacted with prior knowledge such that low-knowledge students benefitted from feedback and 
high-knowledge students did not. This was true regardless of feedback type. In Experiment 2, 
feedback also interacted with prior knowledge on posttest transfer items, such that correct- 
answer feedback promoted transfer in low-knowledge students only. 

The current study contributes to the literature on feedback in at least three ways. First, the 
results are consistent with research showcasing the positive effects of basic feedback for novice 
students’ problem solving (e.g., Bohlmann & Fenson, 2005; Kelly et al., 2013; Luwel et ah, 
2011). Further, they specifically support the use of feedback on computer-based algebra 
homework for middle school students and demonstrate the potential of three feedback types: 
correct-answer feedback, explanation feedback, and try-again feedback. They also show that 
different feedback types can lead to similar outcomes and that providing additional infonnation 
(but see Butler, Godbole, & Marsh, 2013 for a recent counter-example). 

The second contribution is to show that the benefits of feedback are specific to low- 
knowledge students and that high-knowledge students do just as well without feedback during 
problem solving. This is consistent with recent research demonstrating neutral or negative effects 
of feedback on student learning under certain circumstances (Asterhan et al., 2014; Fyfe et al., 
2012; Fyfe & Rittle-Johnson, 2016a; Golke et al., 2015; Krause et al., 2009). These results 
challenge universal recommendations for feedback and suggest that certain students may do just 
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as well without it. For example, high-knowledge students may benefit from solving additional 
problems, rather than using that time to process the feedback message (Hays et ah, 2010). 

The third contribution is to introduce an exciting new method to conduct experimental 
research on feedback and problem solving in an ecologically valid classroom context. The 
ASSISTments project, hosted by Worcester Polytechnic Institute (WPI), is a unique system 
bringing researchers and teachers together to better assist and assess student learning (Heffeman 
& Heffeman, 2014). Many teachers use ASSISTments to assign computer-based homework, and 
researchers are able to write content and run randomized controlled experiments by having 
teachers assign specific problem sets to their students. The key advantage is that the experiments 
are embedded in students’ routine work and so occur within their normal learning environment. 
Heffeman and Heffeman (2014) describe the current study as “the first such trial” of researchers 
unassociated with WPI conducting research using the ASSISTments system (p. 485). 

Future research should continue to examine the effects of feedback on computer-based 
homework. For example, more work is needed to test different types and schedules of feedback 
that are more dynamic and that adjust based on the student response. For example, Bokhove and 
Drijvers (2012) found benefits of fading out the frequency of feedback, such that students first 
receive feedback on each step, then on the exercise as a whole, and then they solve problems 
without feedback. This sort of fading may be particularly important given the results reported 
here and elsewhere that feedback tends to lose its effectiveness once students have sufficient 
prior knowledge in the target domain (e.g., Fyfe et al., 2012; Krause et ah, 2009). 

Future research should also examine the generalizability of the effects over time and with 
different tasks and populations. The current study tested the impact of feedback on a single 
algebra homework assignment for middle school students and assessed within a day or two of the 
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assignment. Certainly, more research is needed with more diverse students from different grade 
levels learning a variety of topics. Further, assessments over weeks and months would be needed 
to know whether these effects of feedback on student learning are maintained at later time points. 
At least one study suggests that feedback continues to interact with prior knowledge and has 
differential effects two weeks after the initial learning episode (Fyfe et ah, 2012). 

Finally, more work is needed to explore ways to enhance the provision of feedback for 
high-knowledge students. One possibility is that high-knowledge students view the feedback 
conditions as less challenging and reduce mindfulness on the task. The idea is that these students 
become less motivated to exert effort on the task because they feel they know how to do them 
and the correct answer will be provided anyway. A potential solution is to use an intennittent 
feedback schedule so that the correct answer is not always provided. Another potential solution 
is to give high-knowledge students more control over the feedback, allowing them to skip 
unnecessary feedback and spend more time on challenge problems. 

In general, the present results highlight the power and variability of feedback effects on 
computer-based homework via the ASSISTments system. Across two experiments, middle 
school students with low prior knowledge benefited from the provision of immediate, corrective 
feedback on an algebra homework assignment, whereas students with higher prior knowledge 
did not. Further, the type of feedback provided was often less important than whether feedback 
was provided at all. On the one hand, researchers and educators can marvel that such minimal, 
basic feedback can improve learning and performance for novice students. On the other hand, 
these results challenge the intuition that feedback is always helpful and suggest that some 
students, under certain circumstances, can do just as well without it. 
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